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Toward standards 

On Analytic Success and Failure 

Martin Petersen 


“Intelligence failure” are two words that analysts 
never want associated with their name. But what is 
an intelligence failure? According to popular culture, 
at least as reflected by the press, our job is to know 
everything and to predict the future — and with a high 
degree of specificity . 1 Intelligence, of course, will 
never be all-knowing and all-seeing, and finished 
intelligence often boils down to informed opinion 
about the implications of precious few facts. This begs 
the real question: what are reasonable standards? 

Setting reasonable standards begins with an apprecia- 
tion of the nature of our work and the field we play 
on. There are three givens in our world, and our suc- 
cess will always be hostage to them. First and fore- 
most, the information available to us is generally 
limited, conflicting, and open to a variety of in- 
terpretations. Time pressures are great, and the issues 
we are asked to address are almost always multi- 
faceted and evolving. The yes or no question is rare; 
we deal in trends and outlooks. The smoking gun of 
an intelligence failure — that piece of reporting that 
points clearly to the truth — is generally visible only 
in hindsight, and sometimes it is not so clear to us as 
it is to our critics . 2 

The policy environment is frequently charged, and at 
times it can be hostile. Not all agendas are above- 
board, and not everyone is interested in an objective 
assessment. For some policymakers, the truth — to the 
degree it can be established — is as likely to make 
them angry as it is to make them free. Thus, intelli- 
gence and the intelligence analyst may be seen as part 
of the problem rather than as part of the solution. 

Finally, a key piece of the puzzle often is what the 
US Government is saying and doing in private, and 
that can be the hardest piece to obtain. The task is 
even more difficult when multiple policy agencies — 
each with its own agenda — are involved. 
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Setting a Standard 

But what standard should we apply to ourselves? If 
we cannot predict the future, is it enough just to 
present the possibilities? Does a policymaker have to 
act on a piece of intelligence in order for us to be 
successful? Does the action itself have to be suc- 
cessful? Can there be an intelligence success and a 
policy failure? 

I believe that in asking the question in this way we 
imply that intelligence, like a game, has a clear out- 
come and that we can keep score of wins and losses. 
The nature of our work indicates otherwise. Rather, 
we should define our standards in terms of the or- 
ganizational purpose of the Directorate of Intelli- 
gence (DI) — support to the policymaker. I believe 
therefore the difference between success and failure 
is whether or not we put the policymaker in a posi- 
tion to make the best-informed decision possible 
under the circumstances: we have to ensure that 
policymakers are given the dynamics of a situation 
or issue, the risks to US interests, the options they 
and their adversaries have, and the consequences of 
various choices. 

To those who might say that this standard is too 
limited and too subjective, I can only answer that, 
however subjective it may be, meeting it is not easy . 3 
Moreover, it does have three concrete elements. We, 
have done our job well if, and only if: 

• We have answered the yes or no questions correctly 
and have gotten the trend lines right. Yes or no 
questions are comparatively rare: Will X invade Y? 
Who will win the election? More frequently, we are 
asked to assess the outlook for, or implications of, 
some development. In such cases, we have to be 
able to say that we foresaw the trend, even if we 
were unable to measure it precisely: growth in 
country X’s exports over the decade; a decline in 
country B’s military capabilities. 
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We have illuminated for the policymaker the The problem frequently is our assumptions about 

forces at work and correctly estimated their im- how things work. Any analyst who has spent time on 

portance, strength, and trends. It is not enough to an issue quickly begins to form assumptions about 

just report or even add context. We have to strive its dynamics. Each of us builds out of assumptions 

to put events in broader perspective and to pro- and opinions a subliminal model about some of the 
vide a framework that allows the policymaker to hardest things to know: motive, character, culture, 

incoiporate new information. We also have to be who has influence, how decisions get made, and the 


able to say that we correctly saw the threats to US 
interests and laid out clearly options and conse- 
quences. We have to state the degree of confi- 
dence we have in our judgments and what factors 
can change them. 

• We have marketed our product in a way that 
allows it to play a role in the policy process. This 
requires us to study our customers, to tailor our 
product so that they can use it easily, and to make 
them aware of the full range of our capabilities . 4 

An intelligence success cannot be declared on the 
basis of a single piece of finished intelligence. 

Rather, success on an issue or set of issues is deter- 
mined over time: did we adjust our analysis as cir- 
cumstances changed; did we effectively target and 
tailor our product to the shifting needs of our con- 
sumers; and were we timely ? 5 If success is always 
somehow conditional, failure can be immediate and 
absolute. It may be a single piece that is poorly 
crafted or the absence of a piece that should have 
been written. In either case, the stain is indelible, 
and no amount of good work before or after com- 
pletely removes it. 

Knowing What You Believe 

Setting a standard is one thing, but meeting it is 
another. I believe the key to getting it right is know- 
ing what you believe and questioning what you be- 
lieve. Knowing what you believe involves identify- 
ing, and separating in your mind, the facts, what is 
not known, your assumptions about how things 
work, and where the wheels can come off. 

As a rule, the facts are not that hard to identify, if 
only because there are generally so few of them. 

And most analysts are good at identifying what it is 
they do not know that is critical to the issue at hand. 
We are also adept at dissecting the facts for clues to 
the unknown. These things are not at the root of 
most flawed analysis. 
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interrelationship of events. This barely conscious set 
of beliefs is the product of everything from earlier, 
accurate analysis to impressions gleaned from snip- 
pets we have read. These assumptions and opinions 
can be, and often are, contradictory. 

Knowing what you believe also involves knowing 
where things can go wrong. Avoiding surprise is 
often a function of knowing what is out there that 
can radically change our assessment. This is more 
than an appreciation for the things we do not know; 
it is an awareness of the potential for the sudden 
so-called low-probability, high-impact event. We also 
need to be attuned to how much our bottom-line judg- 
ment hinges on a particular variable or assumption. 


Three Examples 
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Living With Assumptions 

Because we cannot do analysis without making some 
assumptions, we have to reduce our vulnerability by 
increasing our awareness of our assumptions. Analysts 
have to ask themselves what they believe about the 
priorities, motives, and interrelations involved in the 
issue at hand. The goal is to expose contradictory 
notions and to identify the most questionable as- 
sumptions and the most volatile factors in the ana- 
lytic equation. This process sharpens our thinking, 
thereby making our analysis more persuasive. And, 
most important, it reduces the impact of unspoken, 
half-realized assumptions about new evidence. 

Thinking through one’s assumptions is a tall order 
under the best of circumstances. First, you have to 
set aside the time to do so. Time is the analyst’s 
most precious commodity, but minutes invested in 
thinking about your account on the commute to and 
from the office or while taking a shower are minutes 
well spent. 

Second, you have to limit your thinking to discrete 
problems or issues. You are not going to be able to 
think through all that you believe about China. You 


can, however, think through what you believe about 
China’s relations with Taiwan or the Khmer Rouge’; 
military capabilities or Japan’s approach to GATT 
negotiations. 


Four Traps 

By itself, thinking through assumptions will not en- 
sure accurate analysis. There are many causes of 
flawed analysis, but the four most common are easy 
to avoid. 

Straight Line Projections. One of the easiest things 
to do as analyst is to take the present situation or 
trend and project it. The press and TV commentators 
do this frequently . 6 Straight line projections occur 
when analysts fail to identify and think through all 
the factors driving a problem. The analyst may be in 
over his or her head — or he or she may be guilty of 
sloppy thinking. In either case, the result is an im- 
plicit conclusion that what is true today will be true 
tomorrow. 

This trap has two variants. One is to focus on a sin- 
gle variable and neglect others; polls, numerical 
strength, and rates of growth. The other is to seek a 
precedent in the past and transfer it to the future: the 
government will survive this crisis because it has 
survived similar crises in the past. 

Sometimes the present situation does extend into the 
future, trends continue (that is why they are called 
trends), and historical precedents are valid clues to 
the future. The point is not that such things are al- 
ways wrong. Rather, such things should ring a bell 
that causes us to examine them closely before put- 
ting them in print. 

Seduced by Logic. Much has been written about the 
mirror phenomenon in analysis, the assumption that 
others will behave as we would. We need to remind 
ourselves that risk, goals, honorable outcome, and 
cost/benefit trade-offs are all culture-bound concepts 
that make actions that seem illogical or counter- 
productive to us acceptable to others. But I believe 
this is part of a larger analytic trap — being seduced 
by logic, our principal tool. We forget sometimes the 
importance of emotion and the fact that all leaders 
make bad, even stupid, decisions. 
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Sometimes our analytic ability works against us in 
another way. Analysts by disposition as well as by 
trade, we divine hidden meaning and uncover pat- 
terns in events, even when they are not there. Given 
enough time, a good analyst can fit any two pieces 
of information together. It is important to remember 
that many things are just what they appear to be, and 
that there is such a thing as coincidence. See 
Occam’s razor. 

Leap to Judgment. Because we see making judg- 
ments as one of the things that distinguishes us from 
the press and academics, we sometimes rush into 
them, even when it would be wiser and more helpful 
to the policymaker to reserve our opinion on how 
things will turn out. Coups attempts, elections, and 
negotiations are classic cases for caution, especially 
at the onset. The key variable in any fast-moving, 
evolving situation is that it is fast moving and evolv- 
ing, and we generally are trying to play catch-up. 

We are not forsaking our responsibilities if we with- 
hold a judgment on the outcome when things are too 
uncertain and our sources too limited to support an 
informed opinion. Indeed, we may mislead our con- 
sumers if we try to do so in such circumstances. We 
best serve the policymakers if we inform them about 
the factors at work, the signposts to watch for, and 
the implications of potential outcomes. 

The US Factor. What Washington does or does not 
do is often a key variable. It is essential to cultivate 
policymakers, not only to serve them better but also 
to avoid being surprised by our own government. 

Our analysis is much more likely to be flawed if we 
do not know everything that is causing the other side 
to react. We also have to remember that what the 
others think Washington will do may be more impor- 
tant than what Washington actually does. 

Perceptions, after all, drive one’s behavior. 

The coordination and review process is our best 
defense against these traps, as well as the other things 
we need to keep in mind. When they work as intend- 
ed, these two processes sharpen judgments and help 
us articulate underlying assumptions. They force us 
to test our views and incorporate the expertise of 
others. Managers and analysts alike have a large stake 
in making review and coordination as intellectually 
rigorous (as opposed to mind-numbing contentious) 
as possible. 


Some Reminders 

There is more to successful analysis than avoid- 
ing analytical traps. We would do well to keep 

in mind: 

• How we say things can be as important as 
what we say. No one likes to be told, “I told 
you so,” so be mindful of tone. Our access to 
the policymakers is at their sufferance, and we 
will not be invited back if we insist on putting 
our thumb in their eye at every opportunity. 

• Cynicism is the DI disease. Some policies 
that DIers believe will not work ultimately 
do. We need to remember that the half-empty 
glass is also half full and to look at what is 
going right, as well as what is going wrong. 

• The danger of casual causal judgment. 

Clauses that start with “if” or “given” are 
dangerous when carried over from one piece 
to the next. They postulate a causal relation- 
ship that may no longer be that case, and they 
can easily become .unexamined assumptions. 
Every “if” and “given” should make us 
pause. 

• Numbers are meaningless, unless the reader 
knows how they were derived. Moreover, no 
matter how sound our methodology, there is a 
good chance they are off by some factor. 
Numbers suggest a precision that intelligence 
rarely possesses. 

• Criticism is a given in our business. Indeed, 
it is necessary for success. We can only hope 
that it is honest and informed, which it may 
be more times than we are willing to admit. 

• Our own emotions and biases. Some con- 
sumers will see our work as a report card on 
their performance. We want to make sure that 
that is not our intent. We also have to take 
care that pressure “to get on the team” does 
not lead us to an analytic version of Newton’s 
third law, an equal and opposite reaction. 
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A Final Word 

We are not primarily economists, historians, or polit- 
ical scientists. We are intelligence analysts. It is im- 
portant that each of us makes the time to think about 
our profession — what it is and how to do it better. If 
we do not, we will never be as good at it as we 
could be and as the nation needs us to be. 

NOTES 

1. During the Gulf war, some critics took us to task 
because we could not state exactly how many 
Scuds Iraq had and where each was. Others have 
declared our work on the Soviet Union, Eastern 
Europe, and any number of other places a failure 
because we did not predict a specific event suffi- 
ciently in advance or because we hedged our 
warning. Many of these criticisms start with a 
policy preference and an assumption that the 
Agency has one, too. 

2. In Romeo and Juliet, Shakespeare captured the 
analyst’s point of view best: “Seal up the mouth 
of outrage for a while, til we can clear these am- 
biguities.” 

3. As a quick review of the DI’s Product Evaluation 
Staff’s (PES) studies will reveal, a vigorous PES 
is essential to maintaining standards, however 
they are defined. It is in the best position to 
bring the dispassionate, policy-neutral analysis 
essential to the evaluation of our work. Unlike 
the press, PES also brings an understanding of 
the directorate and its mission on the task. 


4. I sometimes think that the role of the analyst is 
akin to getting policymakers to eat their vegeta- 
bles. We select only the finest vegetables, pre- 
pare them carefully, and serve them attractively. 
The problem is that policymakers also have a lot 
of meat on their plates, and they do not always 
eat their vegetables. Intelligence is good for you, 
but it is not always appetizing. Thus it is that 
you can have an intelligence success and a policy 
failure. 

5. Personal success is something different. To my 
mind, it is more than getting it consistently right. 
As individuals, and as an institution, we ought 

to strive to so establish our reputation for objec- 
tive, insightful analysis that our consumers 
acknowledge — implicitly if not explicitly — that 
they cannot have an informed opinion unless 
they know what we think. They may ultimately 
reject our views, but they nonetheless feel ob- 
ligated to check with us, even when they know 
they are not going to like what they will hear. 

6. Meg Greenfield in the 9 November 1992 issue of 
Newsweek mused on why the press was so often 
surprised and wrong about developments in the 
elections. She argued that the press was swamped 
with data but failed to look beyond them. I would 
add that the press also fell victim to straight line 
projections. The coverage of the primaries, in 
particular, is instructive in this regard. 
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